1. Field of the Invention
The present invention relates to an image processing technique for performing an image restoration process to correct (reduce) degradation of an image produced by an image pickup apparatus.
2. Description of the Related Art
Captured images produced by image pickup apparatuses such as digital cameras include a blur component that is an image degradation component caused by various aberrations of an image capturing optical system (hereinafter simply referred to as “an optical system”) such as spherical aberration, comatic aberration, field curvature and astigmatism. Such a blur component is generated because a light flux emitted from one point of an object forms an image with some divergence on an image pickup surface of an image sensor, the light flux being normally converged at one point if there is no aberration or diffraction.
The blur component herein is optically expressed as a point spread function (PSF), and is different from blur caused by defocusing. Moreover, color blur in a color image generated due to longitudinal chromatic aberration, chromatic spherical aberration or chromatic comatic aberration of the optical system can be said to be a difference between blurring degrees of respective light wavelengths. Moreover, lateral color shift caused by chromatic aberration of magnification of the optical system can be said to position shift or phase shift due to difference in image capturing magnifications for respective light wavelengths.
Fourier transform of the point spread function (PSF) provides an optical transfer function (OTF) showing frequency component information on aberration and being expressed by a complex number. An absolute value of the optical transfer function (OTF), that is, an amplitude component is called a modulation transfer function (MTF), and a phase component is called a phase transfer function (PTF). The MTF and PTF respectively show frequency characteristics of amplitude and phase components of image degradation due to aberration. In the following description, the phase component is expressed as a phase angle by the following expression where Re(OTF) and Im(OTF) respectively represent a real part and an imaginary part of the OTF.PTF=tan−1 [Im(OTF)/Re(OTF)]
As described above, the optical transfer function (OTF) of the optical system provides degradation to the amplitude and phase components of the image, so that the degraded image includes, at each point of the object, asymmetric blur like comatic aberration.
As a method for correcting such degradation of the amplitude component (MTF) and degradation of the phase component (PTF) in the degraded image (input image), there is known a method using information on the optical transfer function (OTF) of the optical system. This method is referred to as “image restoration” or “image recovery”, and a process for correcting the degraded image by using the information on the optical transfer function (OTF) of the optical system is hereinafter referred to as “an image restoration process”. As one of the image restoration processes, a convolution method that performs convolution of an image restoration filter having an inverse characteristic to the optical transfer function on the input image in a real space.
In order to effectively perform the image restoration process, it is necessary to acquire a more accurate OTF of the optical system. For example, information on design values of the optical system makes it possible to calculate the OTF. Moreover, performing Fourier transform on an intensity distribution in a captured image of a point light source makes it possible to calculate the OTF.
The OTFs of general optical systems, except those of optical systems designed and manufactured so as to have extremely high performance, significantly vary depending on image heights (positions in the input image). Therefore, in order to highly accurately perform the image restoration process on the input image, it is necessary to use an image restoration filter produced based on variation of the OTF depending on the image height. In order to change an image restoration characteristic depending on the image height, it is desirable to perform the image restoration process with changing of the image restoration filter in the real space.
Japanese Patent Laid-Open No. 2007-183842 discloses an image processing method that uses a controlling parameter to control degree of the image restoration and thereby continuously changes the degree of the image restoration.
However, depending on characteristics of lenses in the optical system or image capturing conditions, a case may occur in which the MTF falls to zero or approximately zero in a Nyquist frequency band of the image sensor. The fall of the MTF to zero or approximately zero is hereinafter referred to as “zero fall”, and a frequency at which the zero fall occurs is hereinafter referred to as “a zero fall frequency”.
The zero fall is caused by aberration or diffraction. Moreover, the zero fall is also caused by hand jiggling of a user using the image pickup apparatus. FIG. 12A shows an example of the zero fall, where a horizontal axis shows spatial frequency and a vertical axis shows MTF. The zero fall occurs at a frequency indicated by a downward arrow in the figure, and this frequency is the zero fall frequency.
A Wiener filter to be generally used for the image restoration process provides an absolute value of a frequency characteristic (restoration gain characteristic) of the image restoration filter shown in FIG. 12B, where a horizontal axis shows spatial frequency and a vertical axis shows restoration gain. A detailed description of the Wiener filter will be made later.
In FIG. 12B, amplification is obtained in a lower frequency side band than a frequency indicated by a downward arrow, but a low-pass filter effect is obtained in a higher frequency side band than that frequency. This restoration gain characteristic provides an image restoration filter having a profile (coefficients of respective positions (taps) in the filter) in the real space, which is shown in FIG. 12C. The image restoration process using such an image restoration filter generates blur in high frequency details in the input image.
Moreover, FIGS. 13A to 13C show another example. A horizontal axis in FIG. 13A shows frequency up to a sampling frequency that is twice the Nyquist frequency. FIG. 13B shows a restoration gain characteristic of the image restoration filter, and FIG. 13C shows a profile of the image restoration filter in the real space. In FIG. 13C, coefficient values of the image restoration filter are greatly changed at plural positions in a narrow positional range.
There is a case where, when the PSF of the input image is different from the PSF of the optical system because of manufacturing errors of the optical system, luminance saturation in the input image or the like, artifact such as ringing is generated in a resulting image (restored image) of the image restoration process. In such a case, the image restoration filter shown in FIG. 13C becomes sensitive to the difference in PSF, which easily generate the artifact.
Thus, in order to acquire a high-definition restored image, a countermeasure against the zero fall is very important.
Although the image processing method disclosed in Japanese Patent Laid-Open No. 2007-183842 is able to control the degree of the image restoration, this method allows user's change of the degree of the image restoration, but is not able to change the characteristic of the image restoration filter according to the zero fall frequency. Therefore, the method disclosed in Japanese Patent Laid-Open No. 2007-183842 can decrease the degree of the image restoration for suppressing the artifact due to the zero fall, but cannot provide a high-definition restored image.